Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Vinayak/moe final hashem #127

Closed
wants to merge 24 commits into from
Closed

[DO NOT MERGE] Vinayak/moe final hashem #127

wants to merge 24 commits into from

Conversation

carlushuang
Copy link

@carlushuang carlushuang commented Aug 11, 2024 •
edited
Loading

This PR combined vinayak/moe_final with PR:#126, kernel from hashem
Note: to enable triton optimization, need download triton code:

export VLLM_USE_ROCM_CUSTOM_PAGED_ATTN=1
export HIP_FORCE_DEV_KERNARG=1
export VLLM_USE_TRITON_FLASH_ATTN=false

export VLLM_MOE_PADDING=128
export TRITON_HIP_USE_NEW_STREAM_PIPELINE=1
export VLLM_MOE_MFMASWIZZLE=1 
export VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128
# if set VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0, this is just doug + vinayak
export VLLM_SCHED_PREFILL_COUNT=6

you can use benchmark in benchmarks/kernels/benchmark_mixtral_moe_dec.py to do benchmark

# test hashem's kernel with tp-2
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 2
 
# test hashem's kernel with tp-1
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 1
 
# test vinayak's triton kernel with tp-2
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 2
 
# test vinayak's triton kernel with tp-1
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 1

vgokhale and others added 24 commits July 30, 2024 02:54
@vgokhale
Microbenchmark for fused moe
dc5b660
@vgokhale
Revert broken load
d5564d3
@vgokhale
Add persistent kernel from Jingning
1934f71
@vgokhale
Config issues
e01cd34
@vgokhale
Add configs for persistent kernel
24317bb
@vgokhale
Add padding and correct block size
6696142
@vgokhale
Latest perf
077cb78
Merge branch 'main' into vinayak/moe_microbench
eb861ac
@vgokhale
Add masking back
6e70d89
@vgokhale
Add accuracy test
2b94a75
@vgokhale
Merge all changes
c406afc
@vgokhale
Fixes
5b92aa6
@vgokhale
Fix shuffling bug
98a31f2
@vgokhale
Fused moe persistent benchmark
75031c5
@vgokhale
Add test configs for mixtral
7c64459
@vgokhale
Use shuffled layout in UT
b213dfe
@vgokhale
Fix test bugs
7df58ad
@binarman
use mfma 16 for all testcases, reenable moe test [dtype0-2-8-14336-40…
1ca0ad6 
…96-237]
@vgokhale
Enable LDS bypass
0cbe892
@vgokhale
Merge branch 'vinayak/moe_final' of https://github.com/ROCm/vllm into…
f2f1ca5 
… vinayak/moe_final
@vgokhale
Add new configs
c43e3e8
@valarLip
patch wvSpltK_fused_moe from https://github.com/amd-hhashemi/vllm/tre…
ce8a86e 
…e/wvSpltK_fused_moe
@dllehr-amd @valarLip
Add batched prefill via VLLM_SCHED_PREFILL_COUNT
8148b54 
To ensure we we don't run prefills repeatedly during decode, provide a
mechanism to queue up a certain number of prefills before executing.
VLLM_SCHED_PREFILL_COUNT will be the minimum batch count to specify before
executing.  One caveat, the --scheduler-delay-factor should be used to
enforce a longer prefill scheduling value.  This will be set to the value
in VLLM_SCHED_PREFILL_COUNT, if not explicitly provided.  The need for this exists
because an uneven number of prefills can lead to the queue never reaching the
VLLM_SCHED_PREFILL_COUNT.  Causing the server to hang
@carlushuang
add script for decode
b9f05ff
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

@github-actions github-actions bot added the stale label Nov 10, 2024
@github-actions GitHub Actions
Copy link

This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it. Thank you!

@github-actions github-actions bot closed this Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@carlushuang @vgokhale @binarman @valarLip @dllehr-amd